Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis
نویسندگان
چکیده
Some users of social media are spreading racist, sexist, and otherwise hateful content. For the purpose of training a hate speech detection system, the reliability of the annotations is crucial, but there is no universally agreed-upon definition. We collected potentially hateful messages and asked two groups of internet users to determine whether they were hate speech or not, whether they should be banned or not and to rate their degree of offensiveness. One of the groups was shown a definition prior to completing the survey. We aimed to assess whether hate speech can be annotated reliably, and the extent to which existing definitions are in accordance with subjective ratings. Our results indicate that showing users a definition caused them to partially align their own opinion with the definition but did not improve reliability, which was very low overall. We conclude that the presence of hate speech should perhaps not be considered a binary yes-or-no decision, and raters need more detailed instructions for the annotation.
منابع مشابه
How the Spectre of Societal Homogeneity Undermines Equitable Healthcare for Refugees; Comment on “Defining and Acting on Global Health: The Case of Japan and the Refugee Crisis”
Recourse to a purported ideal of societal homogeneity has become common in the context of the refugee reception crisis – not only in Japan, as Leppold et al report, but also throughout Europe. Calls for societal homogeneity in Europe originate from populist movements as well as from some governments. Often, they go along with reduced social support for refugees and asylum seekers, for example i...
متن کاملForced Migration and Global Responsibility for Health; Comment on “Defining and Acting on Global Health: The Case of Japan and the Refugee Crisis”
Forced migration has become a world-wide phenomenon in the past century, affecting increasing numbers of countries and people. It entails important challenges from a global health perspective. Leppold et al have critically discussed the Japanese interpretation of global responsibility for health in the context of forced migration. This commentary complements their analysis by outlining three pr...
متن کاملDefining and Acting on Global Health: The Case of Japan and the Refugee Crisis
What counts as global health? There has been limited discourse to date on the ways in which country-level contexts may shape positioning in global health agendas. By reviewing Japan’s response to the refugee crisis, we demonstrate a clash between rhetoric and action on global responsibility, and suggest that cultural and historical factors may be related to the ways of perceiving and acting upo...
متن کاملSurgical Needs of Refugee Populations in the European ::union::: Implications for Plastic and Reconstructive Surgery
متن کامل
Situation and Text: Representation of Migrants Whilst the Escalation of Refugee Crisis in Great Britain as Compared to Russia
Increasing migration is a vital concern for a globalizing sociocultural environment in today’s world. The UK and developed European countries have become an attractive destination for asylum seekers (labelled as “migrants”) in the past decade. The rapid rise in the number of asylum seekers, which was labelled “migration crisis” (Ruz, 2015), made this topic an integral part of scientific discuss...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید
ثبت ناماگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید
ورودعنوان ژورنال:
- CoRR
دوره abs/1701.08118 شماره
صفحات -
تاریخ انتشار 2017